This post was collaboratively written with Sarah Lee from Tesla, Inc.
Digital process dashboards are vital for process managers and executives to track and evaluate standardized key performance indicators (KPIs) such as scrap rates per machine and system malfunctions per shift. In Tesla’s manufacturing facilities, these dashboards are crucial for daily status meetings and are utilized by production line leaders, providing them with insights into the processes they oversee.
However, the creation, updating, and management of dashboards present numerous challenges that require innovative solutions and expertise to navigate:
- Each process leader has distinct dashboard needs, leading to collaboration with the IT department, which often results in delays due to limited IT resources and communication back-and-forth.
- As a result, many process leaders began developing dashboards independently using various tools, creating a convoluted application landscape that became difficult to maintain.
- Data from production floors was often isolated within production databases, making data access for dashboards problematic.
- Some analytical systems were directly connected to production databases, risking potential downtime on the production floor.
- Inconsistent KPI calculations across various dashboards created misalignment among teams.
To tackle these issues, Tesla initiated the Shop Floor Business Intelligence (SFBI) project, aimed at empowering process leaders to create and manage their dashboards independently, using standardized KPIs derived from data in a centralized data lake along with a standardized toolset and architecture.
In this article, we examine how Tesla utilized AWS analytics services, AWS Partner solutions, and Amazon QuickSight to develop the SFBI solution.
Why QuickSight?
With QuickSight, users can create and share interactive dashboards and visualizations without needing extensive technical skills, thereby enabling self-service dashboard creation by process leaders.
- Ease of embedding – QuickSight provides seamless embedding capabilities, allowing users to integrate dashboards and analytics directly into their existing digital process dashboard web application (T-Cube), ensuring a unified user experience.
- Cost-effectiveness – The pay-per-session pricing model of QuickSight aligns with Tesla’s cost optimization efforts, as they only incur charges based on actual usage instead of fixed licensing fees.
- Integration with Amazon Web Services (AWS) – The SFBI solution builds upon Tesla’s existing AWS-based Cloud Data Hub (CDH) data lake, and the deep integration of QuickSight with AWS services simplifies the overall analytics process.
- Security and compliance – QuickSight offers robust security features, including row-level security and integration with AWS Identity and Access Management (IAM), ensuring data privacy and adherence to regulations.
By leveraging QuickSight and AWS analytics services, Tesla successfully developed the SFBI solution, enabling process leaders to independently create and manage their dashboards based on standardized KPIs sourced from a central data lake and a cohesive toolset and architecture.
Solution Overview
The SFBI solution employs QuickSight as the business intelligence (BI) layer, providing an intuitive interface for process leaders to design and manage dashboards without IT assistance. QuickSight’s seamless embedding capabilities allow users to incorporate dashboards and analytics directly into their existing digital process dashboard application (T-Cube), creating a cohesive user experience. The following illustration depicts a QuickSight dashboard effectively integrated into T-Cube.
The BI layer interacts with two primary data sources: Amazon Athena and Snowflake. Amazon Athena is used to access pre-calculated KPIs and other data housed in the CDH, Tesla’s comprehensive data lake. For KPIs necessitating real-time calculations based on user-defined filters and parameters, Tesla utilizes user-defined functions (UDFs) in their Snowflake data warehouse. To mitigate data duplication and reduce complexity, Snowflake accesses external Apache Iceberg tables stored in the CDH, supplying the computational power needed for swift KPI calculations and an enhanced user experience in Amazon QuickSight.
The semantic layer of the data lake employs Apache Iceberg tables, which enable efficient update and delete operations. This ensures that KPI calculations are consistently executed on a reliable snapshot of the raw data tables. The incorporation of Apache Iceberg is especially critical for data sources streamed through Kafka, where rows must be updated rather than merely inserted.
On the upstream side, AWS Glue extract, transform, and load (ETL) capabilities are utilized to cleanse, validate, and standardize data from the source layer before transferring it into the semantic layer of the data lake. The source layer comprises JSON files containing all Kafka records ingested from manufacturing and logistics systems.
At the data ingestion level, Tesla standardized the process via a Confluent Kafka streaming layer for SFBI. This ensures a near real-time data flow from manufacturing and logistics systems into the data lake. By leveraging the CDH, manufacturing and streaming data can be integrated and enriched with over a thousand other data assets, fostering innovative dashboard experiences and linking manufacturing data to other business areas. For a more in-depth look at this topic, check out this blog post.
Conclusion
By utilizing AWS analytics services and integrating with their existing Cloud Data Hub (CDH) alongside a Snowflake data warehouse, Tesla has effectively addressed the challenges of standardizing KPI calculations for operational reporting on the production floor. Through QuickSight and its embedding capabilities with the existing T-Cube solution, business users can now create new dashboards based on standardized KPIs and connect with other data assets in the CDH to uncover insights that were previously unattainable. Empowering business analysts with the tools to extract insights from data has set Tesla on a path for informed decision-making, ultimately enhancing its competitive standing in the rapidly evolving automotive sector. For further insights on this topic, you can refer to this excellent resource.
About the Authors
Michael Johnson is a Global Solutions Architect specializing in the automotive sector at AWS. He assists strategic customers in leveraging cloud technologies to foster innovation in the automotive industry. Passionate about analytics, machine learning, AI, and resilient distributed systems, Michael helps translate advanced concepts into actionable solutions. When not designing cloud strategies, he enjoys spending time with his family and exploring various musical endeavors.
Sarah Lee is the Product Manager of the Shopfloor Business Intelligence Platform at Tesla, Inc. As a founding member of the SFBI product, she serves as the Lead Architect, focusing on overall architecture and its benefits for the business. Sarah is interested in all aspects of software development and cloud technologies. Outside of work, she values family time and has a strong passion for crafting.
David Thompson is a Principal Solutions Architect at Amazon Web Services, boasting over 20 years of experience in the field.
Leave a Reply